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ABSTRACT 

this report summarizes WLC's 
methodoiogy for determining the 
relative cost effectiveness of 
instructional units in the 
U. S. Naval Academy multimedia 
Leadership Course. Representative 
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effectiveness are reported and 
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I. INTRODUCTION 

This document reports Westinghouse Learning 
Corporation's (WLC) cost and effectiveness experiences 
for the preparation of instructional units in the U. S. 
Naval Academy multimedia Leadership Course. The cost 
collection system is explained, cost experiences and 
instructional effectiveness for various media and 
presentation, forms are reported, and a discussion of 
cost versus effectiveness is presented based on the 
data gained in this project. " ^ 

Over the course of the project, we have taken 
•advantage of several different perspectives in planning 
for cost accounting and its documentation, as reported 
in this document. The skills of project management and 
accounting groups in the New York corporate' offices ' 
have been employed to ensure that corporate cost 
procedures were adhered to, that ease of audit was 
permitted and that newly available techniques were 
incorporated into the system as they applied. 

Major sources of information and guidance for the 
planning' and" preparation of this document have been 
several representative "state-of-the-art" reports that 
examine this area of study. These sources, identified 
in the bibliography, -reflect a. bread range of efforts 
in civilian as well as military instructional 
development projects. 



We have gained much insight into cost effective 
study problems from the full text of a paper, **Ori the 
Bconomic Analysis of Educational Technology" prepared 
in early 1969 for the Indiana State Department of 
Instruction and that state's Tax and Finance Policy 
Commission, by Herbert Kiesling, an economist at 
Indiana University, 
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II. OVERVIEW 

The necessity for cost effectiveness reporting 
stems from the need to weigh the relative merits of one 
medium or presentation form against another, particularly 
when one medium is more or less effective than another. 
When equivalent student performance results can be 
obtained from a variety of media or presentation forms, 
the cost of developing arid revising, after testing, 
becomes a relevant factor of some importance. 

The basic methodology employed in the project 
reported here i^ras (1) the systematic collection of costs 
associated with segment development for each of the 
media/presentation forms; and, (2) the comparison of 
accumulated costs to the relative effectiveness of the 
various segments, as indicated by the performance 

• . ' I ' 

indices. 

This report concerns the costs of developing a 
segment (approximately 50 minutes of student time) of 
instructional materials in various media (video tape, 
audio tape) and presentation forms (linear, branching 
and syndactic programed texts and detailed self-study 
guides) ^ to the point where any given number of copies 



Presentation forms as defined for this paper refers 
to subsets of the printed media. 
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B>iy be reproduced for the course at hand. Development 
costs here begin when reliable terminal and enabling 
objectives and/or a suitable content outline is already 
available. Development costs end. when a completed 
"camera -ready" copy is ready for delivery to the printer, 
or when a "master" video/audio tape is ready for 
quantity duplication. The appropriate Progress Check 
(criterion test) and, where necessary, accompanying 
pahelbooks are considered an integral and necessary part 
of all instructional slgments; therefore, these costs 
arc rightly included as development costs. 

The importance of costing activities requires 
clear statement and understanding of the objective 
desired of the activities. The goal of this paper is 
not to provide a comprehensive and precise profile of 
all of the costs incurred with each of the media 
systems. It is neither feasible within existing 
project constraints , nor is it felt to be necessary to 
prepare a detailed cost specification to reflect total, 
current -^r anticipated expenses for finely-drawn 
activity categories. Thus, another agency is cautioned 
lest this data be used as a definitive predictive 
device for all steps of instruct iona?. development or 
media purchase, installation and operation. 

What is desired, rancher, is what might be 
considered a useful "order-of -magnitude" cost comparison 



for the development of instruction in the selected 
media* In other words ^ it is of greater importance 
for this paper to have a careful compilation of cost 
differences and::^imilarities (in terms of supportable 
extrapolations and approximations) than to have 
complete and documented cost total «; for a wide variety 
of cost categories* 

lu this regard, critical costs must be included 
in the calculations and extrapolations, but it is not 
necessary to include all possible costs* Critical cost 
items include those factors which have a major impact 
on the total "development" costs, which are of interest 
for purposes of subsequent comparison in this report* 
At the same time a number of other items are ignored, 
such as initial specification of objectives, 
administration, staff development, etc. These arc 
without question part of the comprehensive total system 
costs, but are not critical in the terms of this report, 
since chcse factors va>'/ widely in every development 
project. 

The costs reported in connection with the 
development of various media/presentation forms 
represent another standard, added to existing cost 
reports, against which costs incurred by other agencies 
in developing similar instruction can be compared. 



. III. COST COLLECTION SYSTEM 

r 

The present cost accounting system has been 
structured to isolate labor and non-labor costs 
pertinent to both the development o£ the multimedia 
Naval Leadership Course and to the research vhich has 
been conducted during the development and validation 
of the course. Many of the costs accrued during the 
to,tal project are not relevant to the development of 
individual instructional materials alone, but may be 
attributed to such systems efforts as: research, 
.coordination, travel, and the writing of reports not 
directly connected with course development. As such, 
the co,st accounting system is devised to isolate those 
costs which are specific to the multimedia 
instructional system and those which are tangential 
or irrelevant in the actual development. 

The specific objective of this cost accounting, 
in relation to a cost effectiveness study, is to 
provide breakdown costs for preparation of 
instructional segments in a variety of media or 
presentation forms. 

It should be pointed out that although the cost 
accounting system is based on the Course Development 
Model (TP-6.1), there are necessary discrepancies 
between some of the tasks identified in each. In 
general, the course development model isolates groups 



of tasks which may be more inclusive than those 
itemized in the cost accounting system. I^easons are 
that some tasks must be performed in groups to 
initiate subsequent tasks within the same general area. 
The components of the tasks may be of such magnitude 
that they warrant charges against them, and yet they 
derive their importance for the course development 
model only when viewed in combination with other related 
tasks. 

Although specific tasks identified in the course 
development model and cost accounting system may not 
b^^- identical, all of the cost accounting function areas 
are included and recognized in the development models 
The additional function areas specified in the latter 
paper, such. as project management and customer 
interaction, do not carry costing numbers, yet they 
are necessary as links for the other functions. They 
are not, however, considered here since they are 
beyond the scope of this report. Details of the 
collection system for these costs are contained in an 
earlier cost effectiveness report (TP-6.5) dated 
March 7, 1969. 

In the area of developing che instructional 
materials, the costs are first classified according 
to a General Order Number representing gross activity 
groups which correspond roughly to contractual phases 
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and production cycles. 

General Order Numbers : (Note: Early production 
occurred at both Behavior Systems Division, Albuquerque 
and the Annapolis Division) 
BSD ANNAPOLIS 

15031 .15100 Cycle 1: All charges relating to 

materials development prior to the 
first submission for review by USNA. 

15032 15101 Cycle 2: All revision charges 

incurred following the first 
submission of materials to the 
USNA through final acceptance form 
of the primary module in each 
segment by USNA. 

15033 15102 Cycle 3: Production, manufacture 

and revision of presentation 
materials, prior to the spring 
validation period. Revision 
activity charges restricted to 
in-house requirements. 

15034 15103 Cycle 4: Revision activity relating 

to changes required in presentation 
materials by USNA prior to _ 
presentation to Midshipmen. 

15035 15104 Cycle S: All costs relating 

specifically to the validation 
presentation. 
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Cycle 6: All costs relating to 
revision activity resulting from 
the spring validation presentation. 
Production: All costs relating to 
the preparation of cumulative 
posttests (not segment progress 
checks) . 
Not Used 
Not Used 
Not Used ^ 

Production: All charges relating 
to materials development for 
submission to USNA. 
Evaluation and Research: All 
charge ' l..^urred in connection with 
evaluation of data, S-testing, 
reports, and computer services. 
Special Materials: Production, 
manufacture and revision of depth 
core materials, and charges related 
to special media equipment such as 
CBI-70 and COPAR. 
1971 Revision: Revision activity 
relating to changes required due to 
revision of fall run for spring run. 
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15114 Structural Communications: All 



costs relating specifically to 
Structural Communications, 
Budget Center is B 46, 





1511.S 


Not Used. 








Within a given General Order 


function. 


charges 


are 


then further 


separated by specific part and segment 


of 


the instructional system. 






PART SEGMENT 
• 


CODE PART 
POSITIONS 
6 § 7 


SEGMENT CODE 

POSITIONS 
6 § 7 


I 


1 


40 IV 


1 


56 




2 


41 


2 


57 


II 


1 


42 


3 


58 




2 


43 


4 


59 




3 


44 


5 


60 




4 


-4-5 


6 


61 




5 


46 


7 


62 




6 


47 


8 


63 




7 


48 V 


1 


64 




8 


49 


2 


65 




9 


- 50 


3 


66 


III 


1 


51 


4 


67 




2 


52 


5 


68 




3 


53 


6 


69 




4 


54 


7 


70 




5 


55 


8 


71 



9 72 

10 73 
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PART 


SEGMENT 


CODE PART 
POSITIONS 
6 § 7 


SEGMENT 

• 


CODE 

POSITION*? 
6 § 7 


VI 


r 


01 IX 


1 


16 




2 


02 


2 


17 




3 


03 X 


1 


18 




4 


04 


2 


19 


VII 


1 


05 XI 


1 


20 




2 


06 


2 


21 




3 


07 


3 


22 




4 . 


08 XII 


1 


23 




5 


09 


2 


24 


VIII 


1 


10 


3 


25 




2 


11 


4 


26 




3 


12 








4 


13 








5 


14 








6 


15 






Work not 
or task 


assignable 


to a specific 


segment 


00 



All research activities 99 

All work and costs related to administrative 

activities 98 

All work and costs related to administrative 

tests 97 

Instructors Guide activities and related costs 96 

Student Guide activities and related costs 95 
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Next, costs are charged to a specific activity 
■for* or in support of, the production of materials. 



DESIGNATOR 

A 

B 

C 
D 
E 
F 
G 
H 



J 
K 

M 
N 
0 
P 
R 
S 
T 



DESCRIPTION 

Administration 

Analysis -Writing -Behavioral 
Edit-Test Items 

Editorial 

Typing 

Production and Control Administration 

Subject Matter Expert (SME) 

Media Technician (audio visual work) 

Research (media, data processing, 
cumulative posttests) 
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Illustration (layout and art) • 
Secretarial 

S-Testing (validation of materials) 
Programmer - Data Processing 
Systems Analyst - Data Processing 
Statisticifin 
On Site Instructor 
On Site Remedial Tutor 
Printing 

Each activity for every segment is finally assigned 
to a designated cost class. 
DESIGNATOR DESCRIPTION 
A Labor 
B Materials 
C Communications 
D Travel 
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An example will serve to illustrate the manner 
in which all personnel have reported their labor 
and/or expenses for the development of the instructional 
materials . 

Assuming a member of the production group reported 
eight hours of time with the 13 digit sequence: 
1S100071BAB47. Thus - 

15100 designates the function area of materials 

production prior to first submission for USNA 
review • 

07 designates the task of preparing instructional 

materials for Part Seven, Segment 3, specifically. 

1 is a standard WLC separator (digits 2, 3, 4, 
etc. here reflect modules of the segment, 
which constitute research variations of 
identical instructional content). 

B designates the activity: analysis, writing or 
behavioral edit of instructional materials. 

A designates the cost class: Labor 
B47 designates the WLC budget center for Annapolis 
Division. 

All costs connected with the preparation of a 
given instructional segment have been charged against 
a similar 13-digit code, the difference being in the 
activity and cost class designators. For example, if 
the charge is for travel by the media technician, the 



activity and cost class designators would be GD, where 
G represents audiovisual work and D indicates travel. 
If the charge were j£or a necessary reference book or 
other material the designators AB would reflect 
administrative activity (A) and material or service (B) . 

All labor and^nonrlabor detail reports are 
submitted weekly to the WLC computing system. In 
addition to labor itemization, printouts are made of 
specific non-labor charges with respect to the item 
purchased or rented and the draft number of the order. 
In this way, it is possible to verify the applicability 
of the non-labor charge to both the function and task 
area. 

It should be noted that as new tasks are 
identified, they can be added under the appropriate 
function area - up to 99 tasks. Designators will be 
assigned^s new labor and non-labor costing needs arise. 

Another relevant aspect of the cost accounting 
system is the isolation of specific module costs for 
material preparation in media and presentation forms 
dictated by the research design. These costs are the 
specific costs incurred in finalizing materials for a 
module after the content has been specified and analyzed. 
Accounting for module costs has been accomplished by. 
use of the same 13-digit alpha-numeric code. 



15 



For example, the sequence 15102444CAB47 
represents the following: 

15102 Cycle 3: Production of subsidiary module 
presentation materials. 
_ 44 designates preparation of materials for 
Part Two, Segment 3. 
4 designates work for module 4 of the given 

segment (See details below). 
C designates the activity: editing 
A designates the cost class: labor 
B47 reflects the Annapolis budget cente^ 

Segment 3 of Part Two was prepared in four modules 
representing two media forms, with one variation of each. 
Module 1 was a video tape presentation with high response 
demand frequency and covert resonding; module 2: video 
tape with low response demand frequency for overt 
responses; module 3: audio tape and panelbook with low 
resonse demand frequency for covert responses; and, 
module 4: audio tape and panelbook with high response 
demand frequency for overt responses. The variables 
for specific modules were dictated by the research 
design. 

To ensure accuracy in labor and non-labor charges, 
all WLC personnel assigned to the USNA project were 
thoroughly briefed and supervised in the preparation of 
time cards with respect to the appropriate function 
area, task and extension numbers. 
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IV. DESCRIPTION OF MEDIA AND PRESENTATION FORMS 

Instruction for the USNA Leadership Course was 
presented via audio tapes with panelbooks, video tapes 
with panelbooks, linear programed texts, 
syndactic texts, audio tapes with' intrinsically 
programed texts, learning activities summaries, 
and computer-assisted instruction. A description 
of the media and presentation forms follows. 
VIDEOTAPES (VT/PB) 

Videotapes were used to present content in 
lecture format. The lecturer for all video tape 
segments was an active Naval officer with teaching 
background. With the aid of a teleprompter, the " 
lecturer presented all material verbatum from a 
prepared script. 

AH video tapes were prepared at a commercial 
television station. Commercial quality quad video tape 
recorders were used which provided a degree of quality 
unavailable in one inch VTR format. Two cameras were 
used in taping the lecture to allow for integration of 
a series of visuals (charts, photographs, drawings, 
etc.). Additionally, key points of content were super- 
imposed on. the screen during the program. 

At appropriate points in the lecture, the lecturer 
referred the students to a numbered question in the 
panelbook. Students read the question and recorded 
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their answers in the panelbook. Sufficient time was 
allowed for students to respond so they did not have 
to turn off the VTR. The number of questions asked 
ranged from 15 to 22. 
AUDIOTAPES (AT/PB) 

Audiotapes with panelbooks were used to present 
content in lecture format. The lecturer was a 
commercial radio announcer who presented all material 
verbatum from a prepared script. In all audio/video 
research segments > the scripts for audio and video 
presentations were identical. 

All audiotapes were developed in a commercial 
recording facility. In developing the audio 
presentations^ standard recording tapes were used. 
For student use, tapes were transferred to C-60 and 
C-90 cassette cartridges. 

All charts, photographs, drawings, etc., 
accompanying the audiotape lecture were presented in a 
panelbook. (It should be noted that for videotape 
modules, these charts, etc., were presented on the VTR.) 

At appropriate points in the lecture, the 
lecturer referred the students to a numbered question 
in the panelbook. Students read the questions and 
recorded their answers in the panelbook. Sufficient 
time was allowed ^:or students to respond so they did 
not have to turn off the cassette recorder. The number 
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of questions asked ranged from 15 to 22. 
AUDIOTAPE AND INTRINSICALLY PROGRAMED TEXT (AT/ IP) 

As originated by Norman A. Crowder, the intrinsic 
programing technique consisted of routing a student 
through a "scrambled" text on the basis of his response. 
Each response directed him to a different page of the 
text; thus, the student could not read through 
directly and sequentially. 

Combining the intrinsic programing technique 
with an audiotape was a WLC innovation, in this 
teaching mode, the information was presented via the 
tape. While the student listened to it, he also looked 
at a summary page in the text which contained a precis 
of what he was hearing. He then stopped the tape and 
followed the instructions at the bottom of the summary 
page, directing him to a page containing a question 
which tested the information given on the tape and 
summary. Each response to the question referred the 
student to another page which informed him of the 
accuracy of that response. Thus, the student would 
select the alternative which he thought was correct, 
turn to the page indicated for that alternative, 
and find out if he had made a correct selection. If he 
had selected the correct response, 1.5 was instructed 
to go on to another summary page which he read while 
listening to the next audio portion. If his response 
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was partially correct or incorrect, he was either 
told the nature of his error and instructed to proceed 
as described above, or he was instructed to return to 
the summary or question page to study the information 
again and select another alternative. This process 
of interaction between tape and text continued through- 
out the segment. 
SYNDACTIC TEXT (ST) 

A syndactic text is essentially a series of 
linear programed frames each preceded by a brief but 
complete summary of the information presented in the 
frames. Students worked through the syndactic text 
by reading the first summary statement and taking a 
summary quiz of 5 to 8 questions. If the student 
answered all summary quiz questions correctly, he read 
the second summary, took summary quiz 2, etc. 

The student who incorrectly answered one or more 
questions of a summary quiz was required to study the 
linear programed sequence associated with that summary. 
At the end of the programed sequence, the student 
retook the summary quiz. Regardless of his performance 
when he retook the summary quiz, he went on to the next 
summary statement and repeated the procedure. 

Figure A. illustrates student progress through 
the syndactic text. 
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LINEAR PROGRAMED TEXT (LP) 

Linear programed texts used were developed by the 
RULEG and EGRUL methods of programing (Rule -example; 
example-rule). These are essentially programing methods 
of presenting a rule (definition, principle) and having 
the student identify an example of the rule (from 2, 3, 
or 4 choices), or presenting an example and having the 
student identify the rule or principle which is depicted 
in the example. 

Students were instructed to respond to each frame 
by writing their selection (A, B, C, or D) on the 
frame answer sheets. 



LEARNING ACTIVITY SUMMARY (LAS) 

A learning activity summary is similar to a 
traditional textbook or bibliography approach to 
learning. It is a technique very often uied in 
college and graduate seminars to "put the responsibility 
for structuring learning and achieving objectives on 
the student. A learning activity summary was composed 
of three parts: 1) an overview of the segment, 

2) behavioral objectives for the segment, 

3) a bibliography of source matexial that was related 
to each of the objectives. Students worked through an 
LAS segment by reading the overview and objectives and 
studying text materials which were related to each 
objective. T^xt materials were either select pages 

in published text books or supplemental handouts. 

The student studied all text material until he 
felt he had mastered the objectives and could pass the 
progress check. If he did not achieve 80% of the 
objectives on the progress check, he remediated by 
re-studying the prescribed text material. 
COMPUTER ASSISTED INSTRUCTION (CAI) 

Four instructional segments were developed as 
CAI material for the 1500 Instructional System. All 
of the three components of the system (CRT, audio, 
and image projector) were utilized' in the instructional 
units. 



The information frames are presented on the CRT 
screen and image projector. The questions, which 
are often situations in which the student had to decide 
the best course of action, are presented: 1) on the 
audio, where the situation is described; 2) on the 
image projector, where pictures of the situation are 
presented along with the audio; 3) on the CRT screen, 
where the student is asked to select an answer from 
3 to 5 choices. The student's selection, accompanied 
by feedback, is displayed on either the CRT screen or 
the audio tape, and occasionally on the image projector. 
This feedback consists of remediation explaining why 
the selected answer was correct or incorrect. 

The following Outline of Course Structure and 
Media amply portrays the mix of media and presentation 
forms as originally developed for this course of 
instruction. (Note: It should be pointed out that 
the outline sho\m does not indicate media and 
presentation forms for specific instructional units as 
they exist at the end of the project. The most obvious 
discrepancy is the complete lack of videotaped 
instructional units, which have been deleted, for 
various reasons, prior to the end of the project.) 
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TABLE 1 

OUTLINE OF COURSE STRUCTURE AND MEDIA 

Part 
and 
Seg. • 

No. Content Heading Medium 

PART ONE: OVERVIEW OF LEADERSHIP 

1.1 Concepts of Leadership ST 

1.2 Standards of Leadership in the 
Naval Service F-GD 

PART TWO: INDIVIDUAL BEHAVIOR 

27l Introduction to Psychology ST 

2.2 Behavior and its Observation AT- or VT-PB 

2.3 Learning AT- or VT-PB 

2.4 Factors Affecting Learning AT- or VT-PB 
2^5 Attention and Perception AT- or VT-PB 

I 2.6 Motivation ST 

2.7 Conflict ST 

2.8 Neurotic and Psychotic Reactions ST 

2.9 Personality LAS 

PART THREE: GROUP DYNAMICS 

3.1 Characteristics o£ Groups AT- or VT-PB 

3.2 The Relationship of the Leader to 
the Group AT- or VT-PB 

3.3 Group Interactions AT- or VT-PB 

3.4 Conformity as a Factor of Group 
Behavior AT- or VT-PB 

3.5 Relation of the Individual to the 
Group ST 

PART FOUR: ACHIEVING EFFECTIVE 
COMNICATION 

4.1 Importance of Interpersonal 
Communication LP 

4.2 Types of Communication LP 

4.3 The Communication Process 
(Receiver and Barriers) LP 

4.4 The Communication Process 
(Sender and Feedback) AT- IP 

4.5 Formal Communication and Its 
Dimensions AT- IP 

4.6 Informal Communication AT- IP 

4.7 • Communication Under Battle 
Situations AT-IP 
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TABLE 1 Continued 



Part 
and 
Seg. 
No. 



5.1 

5.2 
5.3 
5.4 
5.5 
5.6 
5.7 
5.8 
5.9 
5.10 



6.1 
6.2 

6.3 

6.4 



7.1 
7.2 
7.3 

7.4 

7.5 



8.1 



8.2 
8.3 

8.4 
8.5 
8.6 



Content Heading 

PART FIVE: MILITARY MANAGEMENT 
Introduction to Management and 
the Management Process 
Decision Making and Creativity 
Objectives 



Planning 
Organizing : 
Organizing : 
Organizing: 
Directing 
Controlling 
Coordinating 



Principles and Process 

Structure 

Charting 



PART SIX: AUTFIORITY AND 

RESPONSIBILITY 
Concept of Authority 
Why People Accept/Resist 
Authority 

Delegation of Authority; 
Line-Staff Relationship 
Responsibility 

PART SEVEN: LEADERSHIP BEHAVIOR 

AND STYLE 
Leadership Behavior 
Leadership Style 

Determiners of Leadership Style - 
The Leader 

Determiners of Leadership Style - 
The Group and the Situation 
Participative Leadership 

PART EIGHT: SENIOR- SUBORDINATE 

RELATIONSHIPS 
Organizational Structure § Social 
Distance in Senior-Subordinate 
Relationships 

Officer-Enlisted Relationships 
Assumption of Command and Formal 
§ Informal Leader Relationships 
Introduction to Counseling 
The Counseling Process 
Relations with Seniors and 
Contemporaries 



Medium 



ST 

ST 

ST 

LP 

LP 

LP 

AT- 

AT- 

AT- 

AT- 



or VT-PB 
or VT-PB 
or VT-PB 
or VT-PB 



ST 

ST 

ST 
ST 



AT- or VT-PB 
AT- or VT-PB 

AT- or VT-PB 

AT- or VT-PB 
VT-PB 



LP 

LP 

LP 

LAS 

LAS 

LAS 
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Part 
and 
Seg. 
No. 



9.1 
9.2 



10.1 
10.2 



11.1 
11.2 
11.3 



12.1 

12.2 

12.3 
12.4 



TABLE 1 Continued 



Content Heading Medium 

PART NINE: MORALE - ESPRIT DE CORPS 
Morale VT-PB 
Group Solidarity and Esprit VT-PB 

PART TEN: DISCIPLINE 

Introduction to Discipline AT- IP 

Development and Maintenance o£ 

Discipline AT-IP 

PART ELEVEN: PERSONNEL EVALUATION 

The Role of Evaluation ST 

Enlisted Performance Evaluation ST 

Officer Evaluation ST 

PART TWELVE: APPLIED LEADERSHIP 

Measurement of Effective 

Leadership CAI 

Generally Recognized Characteristics 

of an Effective Leader CAI 

Techniques of Assuming Command CAI 

"That's an Order!" CAI 



a 



ST=Syndactic (multi-level) Text; 

F-GD=Film, Group Discussion; 

AT=Audiotape ; 

VT.=Videotape; 

PB=Pane.lbook; 

LAS=Learning Activities Summary; 
LP=Linear Text; 
IP=Intrinsic Program; 
CAI=Computer Assisted Instruction, 
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V. COST EFFECTIVENESS PRESENTATION 
A. Derivation of Costs Presented > 

There are several general costing guidelines 
that are applicable to all media costs in this report. 
As stated earlier, this report deals with the costs 
o£ developing instructional materials after suitable 
objectives and the scope of the content have been 
settled, and up to the point where a complete "master" 
copy is ready for quantity reproduction. 

It is felt that inclusion of overhead and various 
administrative costs would render a figure of little 
value to those concerned with developing validated 
instructional materials, unless they, too, have 
identical overhead ratios and administrative staff 
and facilities. 

For much the same reason, and based on actual 
experience, the costs of preparing terminal and enabling 
objectives and/or a content outline vary to such a 
degree as to be of little value for future use; The 
delicate negotiation and coordination necessary between 
those persons responsible for course curriculum decisions 
and those persons responsible for materials development 
consumes v/idely varying amounts of time depending, not 
only on the nature of the instructional content, but 
also on the personalities of the personnel involved. 
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The experiences in this project at preparing 
acceptable objectives and content are detailed in the 
final report for the entire project. 

The cost variations experienced in the actual 
production of different quantities of the instructional 
materials themselves prompts the elimination of this 
factor from the compilations shown. During the rather 
long life of the project it was necessary, for various 
restsons, to change the source of reproduction for 
certain materials. Needless to say, the costs involved 
did not remain stable. Equipment pVirchases, installation 
and maintenance, too, varied considerably and have not 
been given consideration here. Decision-makers for 
future activities can quickly and easily determine such 
costs for their particular project from normally available 
sources. 

Time and space constraints prevent reporting 
developmental costs for each media or presentation form 
in the specific, finely-drawn detail that the cost 
collection system permits. The extreme breadth of detail 
available for reporting can be envisioned by considering 
that instructional materials have been developed in 
video tape with panelbook, audio tape with panelbook, 
audio tape v/ith intrinsically programed book, syndactic 
text, linear programed text, and computer -assisted 
instruction (CAI). The research goals and design then 

K 
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forced the preparation of parallel materials to study 
such variables as: high, medium and low response demand 
frequency, high and medium management frequency, overt 
written and overt oral responding and covert responding. 

Cost data is available for every variation, but 
bears little relevance to cost expectations for future 
decisions. Once the "primary" instructional materials, 
in whatever form, have been prepared, the preparation 
of parallel instruction to study research variables 
becomes quite simple and inexpensive. This was the 
strategy of this project: to prepare "primary" 
instructional units first, then the parallel materials 
for research. Data in this report is limited to the 
development of the primary instructional units. 

Not all instructional units are of the same length, 
but vary from approximately thirty-four minutes to just 
over one hour, depending upon presentation form and 
specific content. Therefore, an average cost for each 
developmental function has been derived for each media 
and presentation form as detailed in Section V.B., 
Tables 2 to 8 . The costs attributed to the computer 
programmer for CAI is the average of such costs for each 
of the four CAI units. In the same way, the costs 
attributed to writing or editing or typing a linear text 
is the average o£ costs for that specific function in 
the preparation of all linear programed texts. 
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B. Costs per Media/Presentation Form > 

As stated earlier, costs shown in the following 
tables (2 through 8 ) are limited to averages for the 
development of a primary instructional unit for each 
media or presentation form, excluding specification ' 
of objectives and content, and further excluding 
quantity reproduction for actual student use. Costs 
are grouped according to relevant activities in the 
initial development process: Writer, AV Technician, 
Editor, Subject Matter Expert, related developmental 
production functions, etc. 

Costs peculiar to only one medium, such as: 
video studio and equipment rental and technical 
personnel or computer programmers, are clearly identified 
in the table for the specific medium. 

For convenience, the following definitions have 
been used as necessary in this report: 



VT/PB Videotape with panelbook 

AT/PB Audiotape with panelbook 

AT/IP Audiotape with intrinsically programed text 

ST Syndactic Text 

LP Linear Programed Text 

JiAS Learning Activities Summary 

CAT Computer Assisted Instruction, 
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TABLE 2 

VIDEOTAPE WITH PANELBOOK (VT/PB) 

WRITER - (Script, panelbook quest ionSj 

specifications for illustrations) 500. 

AV TECHNICIAN - CCoordinate art/writing, 

supervise video taping) . 105. 

EDITOR r 25. 

SUBJECT MATTER EXPERT - 25. 

PRODUCTION - 

1. Printed Material 

a. Typist 24. 

b. Proofreader 8. 

c. Artist/Illustrator 100. 

132. 

2. Video Materials 

a. Studio 

(Rent, equipment, personnel, 
Teleprompter script, etc.) 500. 

b. Preparation of slides of 
illustrations 35. 

c. Lecturer/Narrator 25. 

560. 

3. Final Subject Matter Review 15. 

707. 

PREPARATION OF PROGRESS CHECK * 600. 

TOTAL 1,962. 
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WRITER 



TABLE 3 

AUDIOTAPE WITH PANELBOOK (AT/PB) 

• (Script, panelbook' questions, 

specifications for illustrations) 



AV TECHNICIAN 



EDITOR - 



(Coordinate art/writing, 
supervise audio taping) 



SUBJECT MATTER EXPERT - ' 
PRODUCTION - 

1. Printed Materials 

a. Typist 24. 

b. Proofreader 8« 

c. Artist/Illustrator 100. 

2. Audio Materials 

a. Studio (Rent, equipment, 

personnel) 35. 

b. Narrator 50. 

c. Tape Editing ^ 40. 

3. Final Subject Matter Review 
PREPARATION OF PROGRESS CHECK * 



500. 

105. 
25. 
25. 



132. 



125. 
15. 



272, 
600, 



TOTAL 



1,527, 
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TABLE 4 

AUDIOTAPE WITH INTRINSIC PROGRAM (AT/ IP) 

WRITER - (Audio script and complete 

intrinsic program) 750. 

AV TECHNICIAN - (Coordinate art/writing, 

supervise audio taping) 105. 

EDITOR - 75. 

SUBJECT MATTER EXPERT - 40. 

PRODUCTION - 

1. Printed Materials 

a. Typist 60. 

b. Proofreader 40. 

100. 

2. Audio Materials 

a. Studio (Rent, equipment, 

personnel) 35. 

b. Narrator 50. 

c. Tape Editing 40. 

125. 

3. Final Subject Matter Review 20. 

245. 

PREPARATION OF PROGRESS CHECK * gOQ, 

TOTAL 1,815. 
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. TABLE 5 

SYNDACTIC TEXT (ST) 

WRITER - (Summaries, internal quizzes, 
programed sequences) 

EDITOR - 

SUBJECT MATTER EXPERT - 
PRODUCTION - 

1 . Typist 

2. Proofreader 



PREPARATION OF PROGRESS CHECK * 



60. 
35. 



650. 
40. 
40. 



TOTAL 



95. 

600. 
1,425. 



TABLE 6 
LINEAR PROGRAM (LP) 
WRITER - (Complete linear program) 
EDITOR - 

SUBJECT MATTER EXPERT - 
PRODUCTION - 

1. Typist 

2. Proofreader 

PREPARATION OF PROGRESS CHECK * 



24. 
18. 



500. 
40. 
25. 



TOTAL 



42. 
600. 
1,207. 
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TABLE 7 

LEARNING ACTIVITIES SUM?IARY (LAS) 

WRITER - (Research required references, 

prepare guided discussion summary) 100. 

EDITOR - 25. 

SUBJECT MATTER EXPERT • ' 25. 

COPYRIGHT ADMINISTRATION - 

1 . Correspondence 9 . 

2. Copyright Fees 5. 

14. 

PRODUCTION AND ASSEMBLY - 

1. Purchase Required References 

(one each) 40. 

2. Typist 3. 

3. Local Reproduction 

(reference exerpts) 3. 

46. 

FINAL SUBJECT MATTER REVIEW - 12. 
PREPARATION OF PROGRESS CHECK * 600, 

TOTAL 822. 

* A note is in order regarding the uniform cost for 

Progress Checks (criterion tests) for all 
instructional units. All tests were prepared 
directly from the instructional objectives and 
separately from the instructional materials (even 
different writers) to ensure maximum subjectivity 
and to forestall the possibility of "t.^aching the 
.test". Further, all tests are of comparable size 
and format. Costing a single test, then, is 
straightforward and the figure shown ($600. each) 
is the average for all tests, regardless of 
instructional present a ti6n. 



TABLE 8 

COMPUTER ASSISTED INSTRUCTION (CaI) 

WRITER - (Instructional program, audio script, 

specifications for illustrations) 898 

AV TECHNICIAN - (Coordinate art/writing, 

supervise audio taping) . 200 

EDITOR r 105 

SUBJECT MATTER EXPERT - 38, 

PRODUCTION - 

1. Video Materials 

a. Artist/ Illustrator 250. 

b. Film Pack Preparation 508 . 

558. 

2. Audio Materials 

a. Narrator 50. 

b. Audio Pack Preparation 150. 

200. 

3. Computer Programming 1,163. 

4. Final Subject Matter Review 20. 

1,941. 

PREPARATION OF PROGRESS CHECK * 600. 

TOTAL 3,782. 
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Two major factors in the development of 
instructional materials are the writing and production 
functions. The following paragraphs present and 
discuss comparisons of these functions for the various 
instructional forms. 

Examination of the witing costs for the various 
media and presentation forms provides no startling or 
unexpected results to those familiar with or experienced 
at preparing such instructional materials. 

Figure B. compares the costs of writing the 
instructional materials for a single primary unit in 



various forms. 



1,000 
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FIGURE B. 
WRITER COSTS 



898. 



750. 



500. 500. 



650. 



500 



100. 



> 



< 



CO 



CO 

< 



< 



The. equality of cost for preparing the linear 
program (LP) and the scripts and panelbooks for video 
and audio tape presentations stems from the near 
identity of the tasks involved. Each of the three 
forms requires the writer to prepare textual materials 
for the terminal/enabling objectives, interspersed 
with questions (criterion frames) testing the bit of 
instruction just given. While the linear program must 
arrange for a response to every frame, the video and 
audio scripts require specifications for amplifying 
or clarifying illustrations and art work. Apparently, 
orfe requirement nicely balances the other in demands 
on the writer *s time. 

The preparation of the audio tape script with the 
intrinsic program requires the additional care and time 
to write the "scrambled" response choices, to write 
the remediation commonly presented with the various 
incorrect responses, and the detailed, exact cross- 
referencing from audio to intrinsic program and vice 
versa. 

The low cost of writing the Learning Activities 
Summary reflects the simplicity and brevity of writing 
tasks required. The objectives for the unit are stated, 
questions are prepared to guide the student in his 
search for detail and the requisite textual references 
are provided. Simple, straightforward and inexpensive. 



I 

i 38 

i 

i 

Writing the instructional program (as opposed 
to the computer program) for computer-assisted 
instruction makes the greatest demands on the writer. 
True, in a sense, it is an intrinsically programed 
unit, but for the configuration used in this project 
the writer must plan and prepare an audio script for 
the audio mode, a complex intrinsic program for the 
cathode ray tube display and specifications for 
illustrations via the image projector, plus specifying 

\ 

\ some arrangement to accept either constructed or 

selected responses. All must, of course, function in 
perfect unison and sequence to provide for effective 
learning. Quite a challenge to and time consuming for 
even the most highly skilled writer. 

The display of developmental production costs in 
Figure c also holds no mysteries for the initiated. 
In the purely textual forms, as bulk of the text and 
the resulting page count increases, so do the production 
costs. Add the audio medium, and the costs of 
preparing the master tape (studio, narrator) must 
be considered. 
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A sobering fact emerges as one examines the 
iiidividual costs grouped as developmental production 
costs for various media forms. The cost to obtain 
a similar product (one master copy) increases 
dramatically as one moves from audio to video to CAT 
forms. . Eliminating the typist, proofreader, artist/ 
illustrator and final review functions leaves those 
costs peculiar to the given instructional media form. 

FIGURE D. 
MEDIA PRODUCTION AS % OF 

TOTAL DEVELOPMENTAL PRODUCTION COST 

t 

90 




Audio Video CAI 



Thus, production costs peculiar to audio tape 
segments (studio rent, equipment, personnel plus 
narrator plus the editing) represent 4S.5% of the total 
developmental production cost. Production costs 
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peculiar to video tape segments (studio rent, equipment, 
personnel plus illustrative slides plus lecturer) 
represent 80.91 of the total video developmental 
production cost, and a far higher actual dollar cost 
than audio tapes. Production costs unique to CAI 
represent an even higher percentage of its total 
developmental production cost. Preparation of the 
image projector film pack plus the audio tape plus 
the computer programer reaches 861 o£ the total 
production cost. It should be noted that, by itself, 
the computer programer cost for a single instructional 
unit exceeded the combined total developmental 
production costs for one segment each of audio and 
video presentation. 

C. Instructional Effectiveness . 

A cost effectiveness report which merely presents 
cost differentials will be of no value to instructional 
planners and designers, since those instructional media 
or presentation forms which seem extremely expensive 
to develop might actually resull^^ in the highest rate 
and efficiency of learning for all students. Until 
the matter of instructional effectiveness is brought 
into consideration, instructional planners hold an 
unbalanced equation to guide their decisions regarding 
course design and form. Some may consider cost 
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the most critical factor and opt, therefore, for slightly 
less effective student performance. Others may well 
consider effective student learning and end-of -course 
performance to be most critical, at whatever the cost. 
In any event, intelligent planning demands consideration 
of both development cost and instructional effectiveness. 

It is not the intent of this report to present 
conclusive evidence of the merits of one medium and/or 
presentation form over another. Rather, it is intended 
to show the relative efficiency of the various forms. 

It must be made clear, however, that the^ 
statistics shown are not final, end-of -project figures, 
but are limited to results compiled after two complete 
course presentations at the U. S. Naval Academy. The 
first implementation of the total multimedia Leadership 
course occurred during the Spring semester, 1970; the 
second implementation during the Fall 1970 semester. 
The third implementation for the Spring 1971 semester^^ 
continues at this writing; therefore, this data is not 
available. 

Table 9 reports student performance for each 
instructional segment of the course, without regard 
for media or presentation form. The mean of 781 for 
all segments relates quite well with the stated goal 
of 80/80; i.e., at least 801 of the students will 
respond correctly to at least 80% of the criterion 
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measurement items. True, not all segments performed 
as well as desired, however, for a first validation 
field test, the results were not disturbing. 
Instructional materials were revised on the basis of 
careful analysis of empirical data prior to the second 
implementation of the course. The mean for all segments 
86%, is quite satisfactory. More important and note- 
worthy is the dramatic decrease (from 31 to 8) in the 
number of individual segments which did not meet the 
80/80 standard. 

One may still question the eight segments which 
did not perform as expected, but consideration must 
be given to the fact that, for the second implementation 
a number of instructional segments were presented in 
new forms as a result of the deletion of the Learning 
Activity Summaries and the Videotapes. Thus, the 
situation there was one of a first field test, rather 
than a second validation run. 

Naturally, the revision process, based on 
empirical data continued between the second and third 
implementations. Though complete data for the third 
implementation (Spring 1971) is not yet available, 
there are firm indications that the 80/80 standard 
will be attained in all cases. 
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TABLE 9 

Progress Checks - First and Second Implementations 
Mean Percentage Correct, Before Remediation 



Segment 


First 


Second 


Segment 


First 


Second 


Number 


Run 


Run 


Number 


Run 


Run 


1.1 


82 


87 


6.1 


80 


90 


1.2 


67 


84 


6.2 


82 


85 


2.1 


78 


88 


6.3 


91 


88 


2.2 


68 


86 


6.4 


87 


87 


2.3 


65 


70 


7.1 


69 


81 


2.4 


71 


78 


7.2 


83 


87 


2.5 


65 


90 


7.3 


77 


80 


2.6 


71 


82 


7.4 


78 


78 


2.7 


63 


72 


7.5 


74 


83 


2.8 


89 


86 


8.1 


82 


87 


2.9 


65 


90 


8.2 


81 


91 


3.1 


66 


73 


8.3 


92 


92 


3.2 


66 


80 


8.4 


76 


94 


3.3 


75 


70 


8.5 


77 


87 


3.4 


79 


87 


8.6 


77 


87 


3.5 


84 


93 


9.1 


80 


91 


4.1 


83 


92 


9.2 


88 


91 


4.2 


85 


95 


10.1 


93 


95 


4.3 


87 


84 


10.2 


89 


91 


4.4 


92 


95 


11.1 


80 


82 


4.5 


82 


95 


11.2 


91 


93 


4.6 


73 


90 


11.3 


83 


90 


4.7 


77 


90 


12.1 


87 


93 


5.1 


78 


89 


12.2 


86 


82 


5.2 


70 


76 


12.3 


94 


93 


5.3 


74 


92 


12.4 


85 


83 


5.4 


69 


84 








5.5 


59 


82 


MEAN - 






5.6 


70 


82 


ALL 






5.7 


80 


83 


SEGMENTS 


78 


86 


5.8 


90 


90 








5.9 


78 


79 








5.10 


66 


90 









It should be noted that the needs of the students were 
not overlooked at any point in the course. This data is 
indicative of only the first iteration of each instructional 
unit. Any student unable to attain a score of 801 on the 
Progress Check for any instructional segment was immediately 
routed to remediation materials for that segment and retested. 
Students still unable to attain" a score of 80% were then 
routed to individual tutorial sessions to assure mastery 
of the information presented. No student moved ahead until 
all concerned were confident that he had achieved mastery 
of the required learning. 



The effectiveness data shown in Tables 10 through 
16 inclusive, display the average percentage of Progress 
Check (criterion test) items answered correctly by the 
given numbers of students for each segment using a 
specific media or presentation form. Data is sho^m, 
for comparative purposes, for both first and second 
implementations (Spring and Fall, 1970), where 
applicable. The data reflect not only performance with 
the primary instructional unit (the basis for cost 
reporting) for a given segment, but also the performance 
with variant instructional units covering identical 
subject content, which were prepared to satisfy research 
demands . 
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TABLE 10 

AVERAGE % PROGRESS CHECK ITEMS CORRECT, BY SEGMENT WITHIN MEDIA 
VIDEOTAPE WITH PANELBOOK (VT/PB) 

First Imp lementation Second Implementation 

Part § Number Average % 

Segment of PC items 

Number Students correct (NOT 

USED 
BEYOND 
FIRST 

IMPLEMENTATION) 



2.2 


23 


65.0 


2.3 


23 


68 


2.4 


23 


70 


2.5 


23 


62 


3.1 


23 


67 


3.2 


23 


62 


3.3 


23 


74 


X A 


ZZ 


78 


5.7 


22 


72 


5.8 


22 


90 


5.9 


21 


78 


5.10 


22 


63 


7.1 


22 


71 


7.2 


22 


89 


7.3 


22 


71 


7.4 


22 


76 


7.5 


44 


74 


9.1 


44 


80 


9.2 


44 


88 



Unweighted mean I for all segments =73.5 



TABLE 11 

AVERAGE I PROGRESS CHECK ITEMS CORRECT, BY SEGMENT WITHIN MEDIA 
AUDIOTAPE WITH PANELBOOK (AT/PB) 

First Implementation Second Implementation 

Part Q Number Average % Number Average % 

Segment of PC items of PC items 

Number Students correct Students correct 



2.2 


21 


73 


42 


86 


2.3 


19 


64 


42 


70 


2.4 


20 


73 


42 


78 


2.5 


21 


69 


42 


90 


3.1 


21 


66 


42 


73 


3.2 


21 


72 


42 


80 


3.3 


21 


76 


42 


70 


3.4 


21 


81 


42 


87 


5.7 


22 


83 


44 


83 


5.8 


22 


89 


44 


90 


5.9 


22 


77 


44 


79 


5.10 


22 


67 


44 


90 


7.1 


23 


68 


43 


81 


7.2 


• 23 


80 


43 


87 


7.3 


23 


81 


43 


80 


7.4 


23 


79 


43 


78 



Unweighted mean % 

for all segments 74.8 81.0 
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TABLE 12 



AVERAGE 


% PROGRESS 


CHECK ITEMS 


CORRECT, BY SEGMENT WITHIN 


AUDIOTAPE WITH INTRINSIC PROGRAM (AT/IP) 




First Implementation 


Second 


Implementation 


Part § 

Segment 

Number 


Number 
of 

Students 


Average % 
PC items 
correct 


Number Average % 
of PC items 
Students correct 


4.4 




92 


44 


95 


4.5 




82 


44 


95 


4.6 




73 


44 


90 


4.7 




77 


44 


90 


10.1 




93 


43 


95 


10.2 




89 


4J 


91 


Unweighted mean % 
for all segments 


84.3 




92.6 
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TABLE 13 



AVERAGE 




PROGRESS 


CHECK ITEMS 


CORRECT, BY 


SEGMENT WITHIN 


SYNDACTIC 


TEXT (ST) 










First Implementation 




Second 


Implementation 


Part 5 

Segment 

Number 




Number 
of 

Students 


Average % 
PC items 
correct 


Number A-v srage 1 
of PC items 
Students correct 


2.6 




44 


71 




41 


82 


2.7 




44 


63 




42 


72 


2.8 




44 


89 




42 


86 


6.1 




44 


80 




44 


90 


6.2 




44 


82 




44 


85 


6.3 




44 


91 




44 


88 


11.1 




44 


80 




43 


82 


11.2 




44 


91 




43 


93 


11.3 




44 


83 




43 


90 


Unwciighted mean % 
for all segments 


81. 


0 




85.0 
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TABLE 14 

AVERAGE % PROGRESS CHECK ITEMS 

LINEAR TEXT (LP) 

First Implementation 

Part § Number Average % 
Segment of PC items 

Number Students correct 

4.1 44 83 

4.2 44 85 

4.3 44 87 

5.4 44 69 

5.5 44 59 

5.6 44 70 

8.1 44 82 

8.2 42 81 

8.3 44 93 

Unweight " mean 5 

for all gments 78.7 



CORRECT, BY SEGMENT WITHIN MEDIA 



Second Implementation 

Number Average I 
of PC items 

Students correct 



42 


92 


42 


95 


42 


84 


44 


84 


44 


82 


44 


82 


43 


8.7 


43 


91 


43 


92 



87.6 
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TABLE 1 5 

AVERAGE I PROGRESS CHECK ITEMS CORRECT, BY SEGMENT WITHIN MEDIA 
LEARNING ACTIVITY SUMMARY (LAS) 



First Implementation 



Part § 

Segment 

Number 

2.9 

8.4 

8.5 

8.6 



Number Average I 
of PC items 

Students correct 



44 
44 
43 
44 



65 
76 
77 
77 



Second Implementation 

(NOT 
USED 
BEYOND 
FIRST 

IMPLEMENTATION) 



Unweighted mean I for all segments = 73.8 



TABLE 16 



AVERAGE 


% PROGRESS 


CHECK ITEMS 


CORRECT, BY 


SEGMENT WITHIN 


COMPUTER 


ASSISTED 


INSTRUCTION 


(CAI) 




First Implementation 


Second 


Implementation 


Part § 

Segment 

Number 


Number 
of 

Students 


Average I 
PC items 
correct 


Number Average 1 
of PC items 
Students correct 


12.1 


44 


87 


43 


93 


12.2 


44 


86 


43 


82 


12.3 


43 


94 • 


43 


93 


12.4 


44 


85 


43 


83 



Unweighted mean ■ 
for all segments 



88.0 



88.0 



Figure E graphically portrays the learning 
effectiveness of the various media/presentation form 
There is a very important difference between this 
data and that reported earlier. Performance with 
variant instructional units covering identical 
content for research purposes, has been eliminated. 
Thus, the data in the figure has been refined to 
reflect performance only on the primary 
instructional units, and, therefore, correlates 
directly with the cost data reported previously. 



53 



FIGURE E. 



RANGE AND MEAN 'a PC ITEMS CORRECT FOR 
ALL SEGMENTS IN GIVEN MEDIA/PRESENTATION FORM 

(SECOND IMPLEMENTATION - FALL 1970) 
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Table 17 reports performance for individual 
instructional segments grouped by media/presentation 
form. Again, data If restricted to the primary 
instructional units only and excludes the instructional 
variants directed by research design. This data also 
correlates directly with previously reported cost data. 

TABLE 17 

AVERAGE % PROGRESS CHECK ITEMS CORRECT, BY SEGMENT WITHIN MEDIA 
(PRIMARY UNITS ONLY - SECOND IMPLEMENTATION) 



AT/PB 




AT/ IP 




LP 




2.2 


'93 


4.4 


96 


4.1 


91 


2.3 


73 


4.5 


97 


4.2 


94 


2.4 


78 


4.6 


90 


4.3 


84 


2.5 


88 


4.7 


92 


5.4 


84 


3.1 


75 


Mean 


94 


5.5 


89 


3.2 


77 






5.6 


84 


3.3 


72 


ST 




8.1 


83 


3.4 


85 


2;6 


78 


8.2 


91 


5.7 


85 


2.7 


76 


8.3 


96 


5.8 


93 


2.8 


86 


Mean 


88 


5.9 


82 


6.1 


93 






5.10 


91 


6.2 


89 


CAI 




7.1 


79 


6.3 


88 


12.1 


93 


7.2 


92 


11.1 


85 


12.2 


81 


7.3 


85 


11.2 


95 


12.3 


97 


7.4 


81 


11.3 


95 


12.4 


87 


Mean 


83 


Mean 


87 


Mean 


90 
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D . Discussion of Cost versus Ef f ectiveness > 

The data compiled for this report indicate that 
all but one of the media/presentation forms prepared 
for this project can be made effective to the 80V80^ 
standard, and no doubt to a higher standard, given 4 * 
sufficient time, effort and funds. Whether or not 
a given instructional form warrants the expenditure of 
resources required to reach the standard is another 
matter. It is not a matter that will be conclusively 
and categorically decided in this report. 

Useful analysis of cost effectiveness begins with 
a comparison of overall performance of each media/ 
presentation with the respective development costs, as 
in Table 18. Since the development costs are limited 
to those costs necessary to bring instruction to the 
student for the first trial, overall performance for a 
given instructional form is also limited to the results 
of the first course implementation at the U. S. Naval 
Academy. 
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TABLE 18 

DEVELOPMENT COST PER MEDIA/PRESENTATION FORM 
RANKED BY OVERALL PERFORMANCE 



Media 


Overall 


PPT"Fn TTM a Tl ^ Hoir/^l /^r\Tn^-n4- 


CAI 


88 


.0% 


$3782. 


AT/ IP 


84 


.3 


1815. 


ST 


81 


.0 


1425. 


LP 


78 


.7 


1207. 


AT/PB 


74. 


.8 


1527. 


VT/PB 


74. 


.0 


1962. 


LAS 


74. 


0 


822. 



It should be noted that there is no significance 
attached to the fact that the instructional form that 
ranked highest in performance also was the most 
expensive to develop. There was no unusual effort 
devoted or emphasis given to the preparation of the 
computer assisted instruction. The occurence is seen 
as no more than mere coincidence. 

Attention is drawn to the case of the Learning 
Activity Summary as the least expensive form to develop 
and the least effective in performance. As data was 
-compiled and examined, following the first implementation, 
it was decided to delete all LAS units, even though 
that form was the least expensive to prepare. It was 
impossible to visualize the means to improve performance. 
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The LAS units are, by design, quite simple and 
straightforward in presentation form. It was judged 
that, as guidance for student study increases and 
the information presented becomes more refined and 
specific, the actual form of the materials moves 
farther from the Learning Activity Summary design and 
approaches a true programed text. In the situation 
at hand, it would have been fruitless to expend 
additional resources to improve performance with a 
presentation form that permits so few opportunities 
for revision without losing the designed form. Far 
better to discard the LAS form and prepare the content 
in a more auspicious media/presentation, as was done. 

Deletion of the videotape presentations also came 
about as a result of the analysis of data compiled 
after the first implementation of the course concerning 
costs, student attitudes and student performance with 
the media form. 

In the preliminary analysis of variance for the 
pertinent research hypothesis, significant differences 
in performance were found between videotape and audiotape 
instructional units. It could be argued that the 
videotaped instruction could have been effective, had 
the medium been used to its full potential (i.e., if the 
videotape had been largely pictorial rather than being 
specifically restricted to filmed lectures). It is 
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important, however, to point out that videotapes and 
audiotapes were identical in content and presentation 
variables for research; therefore, it is assured that 
no student is deprived of tho opportunity to succeed 
in the course. 

« 

This finding tends to support the WLC hypothesis 
that it is not the media, per se, but the form of 
presentation in the media which is critical to effective 
instruction. 

Another consideration for deletion of videotapes 
was the projected cost and the practicality of having 
so expensive a medium carry sixteen segments of 
instructional material when that same content can be 
presented as effectively in a less expensive mode. When 
large numbers of students are enrolled in the Leadership 
course, there would doubtless be congestion at the 
videotape consoles. 

Another reason was simple cost* Videotapes, as 
they were developed, showed indications of being inferior 
to the audiotapes and were considerably more expensive 
to develop. This coupled with careful estimation of the 
necessary revision costs (estimated at $606. or fully 
33% of the original development cost) brought the 
conclusion that the cost of revising videotapes for the 
second implementation was not justifiable. Since WLC 
was directed to plan and produce the most effective 
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instructional system possible to leave vith the Naval 
Academy at the end of the project, it was deemed to be 
a waste of time and funds to develop a second set of 
videotapes restricted to simple taped lectures. 
Furthermore, future revisions of the tapes by the Naval 
Academy in years to come would also be cost prohibitive 
and their continued use is a matter for some speculation 
at this point. 

A reasonable analysis of cost versus effectiveness 
must also take into consideration the costs and the 
results of the revision activities necessary to achieve 
validated instruction which brings about the desired 
level of student performance. Table 19 compares the 
average percent of increase in performance realized 
after revision of the materials with the costs of the 
revision. Costs are shown as a percentage of the 
original developmental cost, with actual dollar values 
in parentheses. 

TABLE 19 

MEDIA/ PRESENTATION FORMS - RANK ORDER BY PERFORMANCE INCREASE 

Cost 

12.5% C$156.) 
20.0% C$344.) 
14.3% C$206.) 

8.3% C$122.) 
Not Revised. 



Increase 
LP 8.9% 
AT/ IP 8.3% 
AT/PB 6.2% 
ST 4.0% 
CAI 



Direct correlations between the increase in 
performance and the cost of revision is not advisable, 
due to the many factors which determine the require - 
ments for revision and the varieties of activities 
which may be considered as revision activities. 
Good examples are the computer assisted instruction * 
units which incurred continual small costs over a 
period of time as the ^'debugging'' process was carried 
out, yet little activity occurred that could be 
isolated and identified as pure revision on the basis 
of empirical evidence. Generally speaking, however, 
the trend is that revision costs increase as the 
instructional materials are bound to more sophisticated 
hardware . 

A valid conclusion would be that, where course 
design and specific course content are in a state of 
flux, i. e., not firmly established and agreed upon 
by all concerned at the outset, instruction dependent 
upon hardware for presentation should be held to a 
minimum. 
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VI. OTHKR CONSIDERATIONS 
A. Cost Variable?; Not Consideret! . 

A research and development project of the magnitude 
undertaken at the U. S. Naval Academy incurs a vast array 
of costs not detailed or treated in this report. A great 
number of these costs, while an absolute necessity for 
the success of the project, are not pertinent to this 
report, since they arise from activities not critical 
to the purely developmental functions for the 
instructional materials. 

Data processing activities over the three year life 
of the project incurred expenses of over $100,000.00 for 
operational programs and actual machine time. Initial- 
system analysis and design costs are not included in 
that figure. Although can roughly estimate that 
40* of that cost supported strictly research activities 
and the remainder provided the empirical basis for 
materials revision, we have no feel for the accuracy of 
the estimates. A number of machine outputs served both 
research and development processes and cannot be 
accurately charged against one or the other. For the 
future, there is real doubt concerning the necessity for 
sophisticated data collection and processing to verify 
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the continued efficiency or signal the need for revision 
in an established, on-going course of instruction. 

Management costs, too, defy precise categorization. 
All management personnel are professionals attuned to 
the managerial needs of the project and functioning 
primarily in that capacity. As a need arose, however*, 
all managers lent a hand, as necessary, in some other 
skill which is in his repertoire. Thus, the project 
Business Manager, with broad personal experience at 
Naval Leadership, more than once performed as a 
Subject Matter Expert reviewing the instructional 
materials. Other management personnel assisted with 
writing tasks, continuity reviews and so forth, when 
needed. As one found a bit of time free from active 
management functions, another task was taken on for 
whatever time could be allotted to it. Isolating and 
detailing such cost is a practical impossibility. 

A variety of personnel on the staff at the U. S. 
Naval Academy, too, devoted many hours of assistance to 
a large number of project activities, as an adjunct 
to their professional and professorial duties. No way 
has been found to identify and categorize the amounts 
of time so spent, and then isolate and identify the 
cost according to specific media/presentation form or 
to a specific instructional segment. 



The reasons for not treating costs of the type 
briefly touched on above should be obvious. It is a 
certainty that no future large-scale research and 
development project will ever replicate the exact 
conditions surrounding this effort. It is desired to 
provide, to others responsible for instructional 
planning and design, data of validity and relevancy 
to their efforts.. This goal, alone, established 
the limitations for kinds of costs included in this 
report . 

B« Suggested Fut ure Research. 

Past cost-effective studies have considered 
performance in relation to actual and amortized 
costs per media, cost per student hour, per 
specific numbers of students per hour and per given 
numbers of students over many hours. Not surprising 
is the fact that none of the reports known to the 
staff has treated cost effectiveness in a longitudinal 
study to measure effficiency over a protracted time 
span. It is suggested that this is an area for 
fruitful future research. Envisioned is a study of 
cost effectiveness in relation to media efficiency 
and student performance as functions of initial 
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learning, retention over time and practical performance 
in a post- instructional environment. Great value may 
result from such a study, particularly when the 
attitudes, skills and knowledges imparted are directly 
related to the task environment the student meets upon 
course completion. 

Another valuable study would be the cost 
effectiveness of videotaped instruction in the form 
of taped lectures versus a pictorial form which 
utilizes the full potential of the medium. The 
indeterminate results attained in this project with 
taped lectures has not dampened the enthusiasm for 
the medium. Instead, it points rather clearly to the 
need for further research into the potential of 
various presentation forms within the medium. 
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